Systems and methods for synergistic software-hardware power budget management

ABSTRACT

Apparatus, systems, and methods provide an interface between a plurality of hardware resources of a node and a power manager. The interface is configured to define one or more resource groups to expose to the power manager for power measurement and control, assign the plurality of hardware resources to the one or more resource groups, and provide a power allowance to each resource group.

RELATED APPLICATIONS

This application is a continuation of U.S. Non-Provisional patentapplication Ser. No. 14/581,854, filed Dec. 23, 2014, which isincorporated by reference herein in its entirety.

GOVERNMENT INTEREST

This invention was made with Government support under contract numberH98230-11-3-0011 awarded by the Department of Deference. The Governmenthas certain rights in this invention.

TECHNICAL FIELD

This disclosure generally relates to power management for servers andother computing devices.

BACKGROUND

Advances in semiconductor processing and logic design have permitted anincrease in the amount of logic that may be present on integratedcircuit devices. As a result, computer system configurations haveevolved from multiple integrated circuits in a system to multiplehardware threads, multiple cores, multiple devices, and/or completesystems on an individual integrated circuit. As the density ofintegrated circuits has grown, the power requirements for computingsystems (from embedded systems to high performance computing (HPC)systems) have also escalated.

Power and thermal management issues are considerations in designingcomputer-based systems. In HPC systems, for example, a specified thermaldesign power (TDP) and/or the cost of electricity may drive the need forlow power systems. In other systems, including mobile systems, batterylife and thermal limitations make these issues relevant. Optimizing asystem for maximum performance at minimum power consumption is usuallydone using the operating system (OS) or system software to controlhardware elements.

One example of an OS-based mechanism for power management is theAdvanced Configuration and Platform Interface (ACPI) standard. Accordingto ACPI, a processor can operate at various performance states orlevels, namely from P0 to PN. In general, the P1 performance state maycorrespond to the highest guaranteed performance state that can berequested by an OS. In addition to this P1 state, the OS can furtherrequest a higher performance state, namely a P0 state. This P0 state maythus be an opportunistic state in which, when power and/or thermalbudget is available, processor hardware can configure the processor orat least portions thereof to operate at a higher than guaranteedfrequency. As discussed below, however, such solutions are sub-optimalin handling performance variations across collaborating threads onsystems in which the objective is to have multiple threads complete aparticular task (or a task phase) at the same time.

BRIEF DESCRIPTION OF THE DRAWINGS

Arrangements and embodiments may be described in detail with referenceto the following drawings, in which like reference numerals refer tolike elements.

FIG. 1 is a block diagram of a system according to certain embodiments.

FIG. 2 is a block diagram of a CPU socket according to an exampleembodiment.

FIG. 3 is a block diagram of a CPU socket according to another exampleembodiment.

FIG. 4 is a flow chart of a method for managing power in a nodeaccording to one embodiment.

DETAILED DESCRIPTION OF EMBODIMENTS

Embodiments disclosed herein are generally directed to an apparatus,method, and other techniques for managing workloads on a computersystem. Certain embodiments provide an interface between a processingnode or device (e.g., a central processing unit (CPU)) and a softwareagent or apparatus that provides power management. The interfaceprovides flexible power management (e.g., controls and measurements).The interface allows, for example, controlling power and performanceacross arbitrary groups of computing threads and resources within aprocessing node in an efficient and scalable manner.

An application executed by a computer system may include a group oftasks that work together toward a common purpose (e.g., computing asolution to a problem). The tasks may execute on one or more nodes.Inside a node, there may be one or more tasks, and each task may includeone or more processing threads. The execution time of an application orsystem behavior may be divided into segments that exhibit a stable orpredictable operational behavior. The stable or predictable segments arereferred to herein as “phases” or “task phases” of an application. Eachof the phases comprises, for example, a segment of the application thatexhibits operational behavior different than that of the other phases. Aphase may include, for example, a compute bound phase, a memory boundphase, a communication bound phase, a mixed compute/memory/communicationphase, an idle phase, an active phase, and/or another phase bound to anyoperational behavior.

Processing nodes with a large number of computing threads expose achallenge to software load balancing processes because performance canvary substantially due to diverse conditions across threads including,but not limited to parameters such as cache miss rate, translationlookaside buffer (TLB) miss rate, or branch misprediction rates.Manufacturing variations can also expose performance differences whenoperating at non-guaranteed frequency points (commonly referred asTurbo) or under thermal or power constrained conditions.

To compensate for such natural variations, certain embodiments include ahardware-software interface that enables runtime dynamic biasing ofpower allocation across a processing node's resources to regulaterelative performance. The interface may allow allocations to bespecified using a “universal coin” that enables deterministic behaviorwhen changing allocations, as long as other conditions that might impactperformance in the system such as thermals, socket power limit, andruntime application behavior remain constant.

By way of contrast with the disclosed embodiments, other solutions oftenincrease performance overhead, do not scale for processing nodes with alarge number of computing threads, and do not provide a consistent andunified way to express dynamic power and performance tradeoffs. Othersolutions may enable different logical threads to express differentlevels of performance requirements in a processing node. As discussedabove, for example, legacy APIC P-State interface enables systemsoftware to specify a concrete operating frequency request for eachthread. Some specifications also comprehend the concept of abstractedperformance states that can be requested by system software andinternally translated by hardware into specific voltage-frequency pointsdepending on architectural and dynamic operating conditions.

However, such solutions are sub-optimal in handling performancevariations across collaborating threads on systems in which theobjective is to have multiple threads complete a particular task (or atask phase) at the same time. This type of software is commonly used,for example, when decomposing large complex problems into smaller blocksof computation on scientific and enterprise applications including butnot limited to traditional high performance computing (HPC) workloads.

Operating systems that are commonly used to control P-States do notcomprehend user-level load balancing information. Such information maybe used, for example, to understand which configuration provides optimalrelative performance across computing threads at a particular point intime in order to improve or optimize the execution time for a task or atask phase. P-state interfaces expose per-thread control knobs to systemsoftware. While such approach provides flexibility, it also leads tosubstantial overhead as each computing thread generally needs to beinterrupted in order to update the performance-level request. Moreover,the instructions used to modify such parameters are often protected andcan only be executed by system code, which means that each threadundergoes a transition from user-level to system-level execution mode.Such an approach presents both a spatial and a temporal scalabilityproblem. Certain embodiments disclosed herein, however, allow forfrequent and rapid adjustment of power and/or performance biasinformation on systems with a large number of hardware threads.

Some server systems operate in power-constrained conditions that make“power allowance” a natural unit of management across nodes andcomputing threads within a node. Certain server systems, however, usedisjointed mechanisms with different driving software and/or firmwarecomponents for limiting power on a particular computing node, andspecify relative performance across threads like the aforementionedP-state interface. Embodiments disclosed herein, however, provide acentralized control interface that enables a single thread in aprocessor node, such as a single die processor socket including multiplecores, to bias power utilization across computing threads and othercomponents within such nodes through the manipulation of a set ofcontrol registers with scalable granularity.

Various embodiments also relate to an apparatus or system for performingthe operations disclosed herein. The apparatus may be speciallyconstructed for the required purpose or it may comprise ageneral-purpose computer as selectively activated or reconfigured by acomputer program stored in the computer. The procedures presented hereinare not inherently related to a particular computer or other apparatus.Various general-purpose machines may be used with programs written inaccordance with the teachings herein, or it may prove convenient toconstruct more specialized apparatus to perform the disclosed methods.The required structure for a variety of these machines will appear fromthe description given.

Reference is now made to the drawings, wherein like reference numeralsare used to refer to like elements throughout. In the followingdescription, for purposes of explanation, numerous specific details areset forth in order to provide a thorough understanding thereof. It maybe evident, however, that the embodiments can be practiced without thesespecific details. In other instances, well-known structures and devicesare shown in block diagram form in order to facilitate a descriptionthereof. The intention is to cover all modifications, equivalents, andalternatives consistent with the claimed subject matter.

FIG. 1 is a block diagram of a system 100 according to certainembodiments. As shown in FIG. 1, the system 100 includes a processingnode 105, which may be a single die processor socket including aplurality of cores 110 a-110 n of a core domain. In other examples, theprocessing node 105 may be a server or a computing system component suchas a microprocessor, an individual core of a multi-core microprocessor,a memory controller hub (MCH), an input/output controller hub (IOH), amemory device, a network interface, or any other type device having oneor more resources or power domains that are capable of being monitoredand controlled for power consumption. The term power domain, or simplydomain, is used herein to refer to any component or set of componentscapable of being monitored and controlled for power consumption.

The various cores 110 a-110 n may be coupled via an interconnect fabric115 to a system agent or uncore domain 120 that includes variouscomponents. As shown, the uncore domain 120 may include a shared cache130 that may be a last level cache. In addition, the uncore domain 120may include an integrated memory controller (IMC) 840, variousinterfaces (IF) 150 a-150 n, and a power control unit 155. A powerdelivery system 170 is coupled to the processing node 105 and mayinclude a switching or linear voltage regulator (VR, not shown) capableof supplying a range of voltages to the processing node 105. Theprocessing node 105 may communicate with a system memory 160, e.g., viaa memory bus. In addition, by interfaces 150 a-150 n, connection can bemade to various off-package components such as peripheral devices, massstorage and so forth. While shown with this particular implementation inthe embodiment of FIG. 1, the scope of the present disclosure is notlimited in this regard.

The system 100 further includes a power manager 180 and a power controland measurement interface 182. The illustrated power manager 180 may bea third party element or included in the power delivery system 170, andmay be implemented in fixed functionality hardware, microcode, firmware,software, or any combination thereof. In other embodiments, the powermanager 180 may be included in or executed by the processing node 105.For example, the processing node 105 may include one or more processingcores configured to implement the power manager 180.

The power control and measurement interface 182 provides a flexibleinterface between the power manager 180 and the processing node 105. Thepower control and measurement interface 182 provides for an arbitrarygrouping of computing threads and resources in the processing node 105.Thus, the power manager 180 may be exposed to a variety of differentpower domains or sub-domains in the processing node 105. For example,the power control and measurement interface 182 may selectively groupany combination of one or more of the cores 110 a-110 n with theinterconnect fabric 115, shared cache 130, IMC 140, one or more IF 150a-150 n, power control unit 155, or any other core or uncore element.Thus, each selectively combined group can be managed as a respectivepower domain.

Although the power control and measurement interface 182 is shown as aseparate device or module, the power control and measurement interface182 may be part of or executed by the power manager 180 or theprocessing node 105. The power control and measurement interface 182 maybe implemented in fixed functionality hardware, microcode, firmware(e.g., run on the power control unit 155), software, or any combinationthereof.

In the example embodiment shown in FIG. 1, the power control andmeasurement interface 182 includes a power monitor module 184, aperformance controls module 186, a power enforcer module 188, and apower budget manager module 190. The power monitor module 184 isconfigured to measure the power consumption of each selected group ofcomputing threads and resources in the processing node 105, and toprovide corresponding power monitoring data to the power enforcer 188,the power budget manager 190, and the power manager 180.

The performance controls module 186 includes controls to adjust theperformance and power of each selected group of computing threads andresources in the processing node 105 through, for example, adjustingpower, adjusting frequency, throttling bandwidth, and/or otherperformance parameters.

The power enforcer module 188 configures the performance controls module186 and adapts the configuration over time to maintain an overallaverage power over a programmable time window. The power enforcer module188 comprises a control system that may be implemented with standardcontroller techniques. The power budget manager module 190 compriseslogic that outputs power budgets to the power enforcer module 188. Thepower budget manager module 190 may provide several different interfacesto the power manager 180 that enable it to either directly specifyspecific power budgets for each selected group of computing threads andresources in the processing node 105 (e.g., in units proportional toWatts) or to specify budgets indirectly by specifying hints aboutrelative importance or influence of each selected group of computingthreads and resources in the processing node 105 (in abstract units) andletting the power budget manager module 190 apply internal heuristics tothe hints and map them to specific budgets for the selected group ofcomputing threads and resources in the processing node 105.

In certain embodiments, the power budget manager module 190 includescontrol registers 192 configured to define resource groups and powerallowances for each resource group. The granularity at which power iscontrolled is adjustable by not forcing a particular mapping ofresources on each of the defined groups. Rather, the resources may bedynamically selected. As discussed in detail below, the controlregisters 192 include resource group control registers 194 configured todefine the resource groups, and power allowance control registers 196configured to define a power allowance or limit for each resource group.

Example embodiments discussed below are directed to a single die centralprocessing unit (CPU) socket including multiple cores and otherintegrated components. In certain such embodiments, only one instance ofthe control registers is implemented on each CPU socket, and the valuesstored in the control registers affect the behavior of all managedresources within the CPU socket. Those skilled in the art will recognizefrom this disclosure, however, that the embodiments shown in the CPUsocket embodiments may be applied to many different types of computingdevices that comprise a plurality of resources that may be selectivelycombined and managed.

FIG. 2 is a block diagram of a CPU socket 200 according to an exampleembodiment. The CPU socket 200 includes a plurality of manageableresources 210. In this example, the plurality of manageable resources210 includes a plurality of execution cores 212, an interconnect fabric214, and a plurality of input/output devices 216. The CPU socket 200further includes four resource group control registers 218 and fourpower allowance control registers 220. In other embodiments, any numberof resource group control registers 218 and power allowance controlregisters 220 may be used, based on a maximum number of groups to defineand/or a maximum number of manageable resources 210 to include withindefined groups.

System software or an agent (e.g., the power manager 180, power budgetmanager 192, and/or the power control unit 155) may write values to theresource group control registers 218 to define an arbitrary number ofgroups of hardware resources to be managed, up to a maximum number ofgroups determined by the total number of resource group controlregisters 218 and the addressing space used. The greater the number ofgroups defined, the more fine-grained the power control granularity onthe CPU socket 200. However, a tradeoff may exist wherein a greaternumber of groups defined by the resource group control registers 218 mayresult in a higher implementation complexity and cost.

In the example shown in FIG. 2, up to four groups may be defined (Group0, Group 1, Group 2, and Group 3). During operation, the resource groupcontrol registers 218 may define less than four groups. The systemsoftware or agent may, for example, dynamically change between definingzero, one, two, three, or four groups. When no groups are defined, theCPU socket 200 may be responsible for determining power allocationsamong the manageable resources 210 (e.g., as determined by the powercontrol unit 155 shown in FIG. 1). The resource group control registers218 are shown in FIG. 2 as Group Definition Register 0 configured todefine Group 0, Group Definition Register 1 configured to define Group2, Group Definition Register 2 configured to define Group 2, and GroupDefinition Register 3 configured to define Group 3.

The resource group control registers 218 may be modified dynamically bysystem software or another agent at any time during operation to allowfor adaptive policies as needed due to changing workload characteristicsor application operational behavior, as well as to different systemmanagement policies. In the example shown in FIG. 2, the resource groupcontrol registers 218 are currently configured to assign a first set ofexecution cores 212 (sixteen shown) to Group 0, a second set ofexecution cores 212 to (twelve shown) to Group 1, the interconnectfabric 214 to Group 2, and the plurality of input/output devices 216(three shown) to Group 3. At any time during operation, different valuesmay be written to the resource group control registers 218 to assign anyof the manageable resources 210 to any group in any combination,including combinations of core and uncore resources within the samegroup.

In one embodiment, each resource group control register 218 isconfigured to enumerate one or more of the manageable resources 210 thatbelong to its corresponding group using an unsigned integer resourceidentification on a design specific addressing space. In suchembodiments, each of the manageable resources 210 to be controlled isassociated with a unique identification or “address” within the CPUsocket 200. The value zero may be reserved to denote no resource (i.e.,the particular resource group control register 218 is not currently usedto define a group).

The power allowance control registers 220 are each associated with arespective group and are shown in FIG. 2 as a Group 0 Power Allowanceregister, a Group 1 Power Allowance register, a Group 2 Power Allowanceregister, and a Group 3 Power Allowance register. During operation, thesystem software or other agent may write values to each of the powerallowance control registers 220 to specify a selected power allowance orlimit for its respective group. In certain embodiments, the CPU socket200 is configured to distribute power evenly among the manageableresources 210 assigned to each group. In other embodiments, the CPUsocket 200 may be configured to selectively allocate the power among themanageable resources 210 in each group, as long as the total powerconsumed by the manageable resources 210 does not exceed the powerallowance or limit indicated by the power allowance control register 220corresponding to the group.

FIG. 3 is a block diagram of a CPU socket 300 according to anotherexample embodiment. The CPU socket 300 includes a plurality ofmanageable resources 310. In this example, the plurality of manageableresources 310 includes a plurality of execution cores 312, aninterconnect fabric 314, and an integrated network interface controller(NIC) 316. Each of the execution cores 312 may support multipleexecution threads. Each of the manageable resources 310 is configured tobe managed (e.g., exposed to measurement and control) from a powerand/or performance point of view. The CPU socket 300 further includesthree resource group control registers 318 and three power allowancecontrol registers 320. The embodiment shown in FIG. 3 is provided toshow a simplified example of how an addressing space may be used withthe resource group control registers 318 and the power allowance controlregisters 320 to provide a hardware/software interface. However, thoseskilled in the art will recognize that the example shown in FIG. 3 maybe scaled for any number of manageable resources 210, resource groupcontrol registers 318, and power allowance control registers 320.

In the example shown in FIG. 3, up to three groups may be defined (Group0, Group 1, and Group 2). Unlike the example shown in FIG. 2, theexample shown in FIG. 3 does not show any particular grouping ofresources. Rather, the solid and dashed lines from the resource groupcontrol registers 318 represent the ability to assign any of themanageable resources 310 to any of the groups. During operation, theresource group control registers 318 may define less than three groups.The resource group control registers 318 are shown in FIG. 3 as GroupDefinition Register 0 configured to define Group 0, Group DefinitionRegister 1 configured to define Group 2, and Group Definition Register 2configured to define Group 2. The power allowance control registers 320are each associated with a respective group and are shown in FIG. 3 as aGroup 0 Power Allowance register, a Group 1 Power Allowance register,and a Group 2 Power Allowance register.

In the example shown in FIG. 3, the manageable resources 310 comprisefourteen resources including twelve execution cores 312, theinterconnect fabric 314 across the execution cores 312, and the NIC 316.The example also uses a 4-bit addressing space to identify up to threegroups of resources (Group 0, Group 1, and Group 2). The groups can beassigned any of the manageable resources 310 in the CPU socket 300 inany combination. In this example, each of the resource group controlregisters 318 comprises a 64-bit register to allow enumeration of allpossible combinations of the manageable resources 310 within each group.Thus, if RESOURCE_GROUP_G identifies a register corresponding to aparticular group (where G=0, 1, or 2 in this example), andResourceID_(R) identifies a particular resource (where R=0 to 15corresponding to the 4-bit address space in this example), then theresource group control registers 318 in this example may be defined asfollows:

RESOURCE_GROUP_0[3:0]: ResourceID₀ . . . .

RESOURCE_GROUP_0[63:60]: ResourceID₁₅

RESOURCE_GROUP_1[3:0]: ResourceID₀ . . . .

RESOURCE_GROUP_1[63:60]: ResourceID₁₅

RESOURCE_GROUP_2[3:0]: ResourceID₀ . . . .

RESOURCE_GROUP_2[63:60]: ResourceID₁₅.

By way of example, it is assumed that the system software or other agentdefines the relative power allowance for execution cores 1, 2, and 12 inGroup 0 versus the power allowance for execution cores 3 to 11 in Group1, and the communication sub-systems (i.e., the fabric interconnect 314and the NIC 316) in Group 2. In this example, the execution cores 312are respectively assigned identifiers 1 to 12 in the address space, theinterconnect fabric 314 is assigned identifier 14 and the NIC 316 isassigned identifier 15. To express the selected groups, the systemsoftware or other agent writes to the resource group control registers318 as follows (where, in the identified register, [3:0] indicates bits0 to three, [7:4] indicates bits 4 to 7, [11:8] indicates bits 8 to 11,and so forth):

RESOURCE_GROUP_0[3:0]=1; [7:4]=2; [11:8]=12

RESOURCE_GROUP_0[63:12]=0

RESOURCE_GROUP_1[3:0]=3; [7:4]=4; [11:8]=5

RESOURCE_GROUP_1[15:12]=6; [19:16]=7; [23:20]=8

RESOURCE_GROUP_1[24:21]=9; [28:25]=10; [32:29]=11

RESOURCE_GROUP_1[63:33]=0

RESOURCE_GROUP_2[3:0]=14; [7:4]=15

RESOURCE_GROUP_2[63:8]=0.

The power allowance for each of the groups is defined by the powerallowance control registers 320, as described below. In this example,power is assumed to be evenly distributed across resources within agiven defined group by the underlying hardware implementation. Asdiscussed above, however, other embodiments may allow the CPU socket 300to selectively distribute the allocated power across the resources ineach group.

Continuing with the example shown in FIG. 3, the system software oranother agent may write values to the power allowance control registers320 to specify a selected power allowance for a particular resourcegroup. The power allowance in this example is defined as a unit-less8-bit encoded natural value ranging from 0 to 255. A zero value writtento a power allowance control register 320 may be used to express nopower limit for a particular resource group. In such embodiments, a zerovalue indicates that the hardware (e.g., the CPU socket 300) isresponsible for deciding on the power for the particular group withoutinstructions from the system software or other agent.

A non-zero value written to a power allowance control register 320 maybe used by the CPU socket 300, in certain embodiments, as a relativepower bias indication as compared to other non-zero power biasindications. In other words, assume that S_(p)(t) denotes the totalsocket power allowance at a given time t and that PA(g) denotes an 8-bitquantity expressed on power allowance control register 320 for a groupg; the requested power allowance for a group g at a given moment t maybe expressed as

${{P\left( {g,t} \right)} = {{{Sp}(t)} \times \frac{{PA}(g)}{\sum\limits_{i = 0}^{N - 1}\;{{PA}(i)}}}},$where N represents the total number of resource groups defined.

Following the same example shown in FIG. 3 of a twelve core socket withthree groups, the system software or other agent may define thefollowing allowance values to request devoting 20% of the CPU socket'spower limit to cores 1, 2 and 12 in Group 0; 50% of the CPU socket'spower limit to the remaining cores in Group 1, and 30% of the CPUsocket's power limit to the communication sub-systems (i.e., the fabricinterconnect 314 and the NIC 316) in Group 2. Thus, if the powerallowance control register 320 comprises a single 64-bit registeridentified as POWER_ALLOWANCE wherein bits 0-7 correspond to Group 0,bits 8-15 correspond to Group 1, and bits 16-23 correspond to Group 2,then data may be written to the power allowance control register 320 asfollows:

POWER_ALLOWANCE[7:0]=20; [15:8]=50; [23:16]=30

POWER_ALLOWANCE[63:24]=0.

In certain embodiments, the value written to each power allowancecontrol register, or (in the example above) in defined portions of asingle power allowance control register, does not indicate a specificpower value (e.g., in Watts) corresponding to a specific group ofresources. Rather, as illustrated in the example above, the value (e.g.,an unsigned integer) may serve as an abstract “universal coin” to managepower across resources using the software layer. Increasing a powerallowance in one resource group by a value “P” and decreasing a powerallowance in another resource group by the value “P” is power neutralwithin a node (e.g., the CPU cores 200, 300 in the above examples), atleast from the perspective of a power manager's ability to accuratelymeasure and control the node's power. Increasing the power allowance fora particular resource group may translate into increased performance forthe particular resource group (and possibly the node), and may beselected so as to not limit the node's performance when other relevantparameters are unchanged.

In certain embodiments, using a “universal coin” as an abstract yetconsistent management unit removes the need for software to understandthe mapping of an implementation unit to actual power distributionwithin the node, which may enable smaller guard bands. Using a“universal coin” may also remove the need for software to understandwhat is being adjusted within the node (e.g., change in voltage and/orfrequency, change in link width, etc.). Thus, no unit equivalence (e.g.,Watts, Volts, Hertz, etc.) is implied or required across differentproducts implementing the interface.

In certain embodiments, the hierarchical nature of the interface enablescooperative control systems. For example, a first control system may seta power budget comprising a total power limit across all of the groups,and a second control system may rebalance the power budget between theresource groups without the need to know the total power limit.

In certain embodiments, the encoding for group's power allowanceprovides an interface that is unit-less and self-adapting to the numberof resource groups supported by a particular implementation. Further,the total number of bits used to describe the power allowances naturallyincreases with the number of supported groups, which avoids encodingaccuracy problems when a large number of groups is defined.

The encoding cost in bits for systems with a large number of resourcesto be controlled may grow considerably when combined with the supportfor a large number of groups. For example, assuming that Q representsthe maximum number of resources to be managed, G represents the numberof groups supported, and M represents the maximum number of resourcesthat can be enumerated on each group, the total storage cost per nodefor both power allowance and resource group registers is(8×G)+log₂(Q+1)×M×G bits.

Other embodiments may use other schemes to encode a group's powerallowance. For example, a degenerated case may be considered for thescenario in which a large number of resources needs to be managed whileonly two groups are used. For such an embodiment, a more efficientencoding may be used by defining which group each resource belongs towith a single bit and using the position in the control register as animplicit resource identifier. For example, a single resource groupcontrol register may be used to combine cores 0 and 1 into a first groupand cores 2 to 63 into a second group, in a 64-core hypothetical systemsuch that:

RESOURCE_GROUP[63:0]=0x3

Thus, using such encoding may reduce the overall cost for G=2 groups to(8×G)+Q total bits.

Certain embodiments provide a hardware-software synergy for power budgetmanagement. For example, the interface disclosed herein allows anarbitrary number of resource groups to be defined and also provides theflexibility to define power allowance hints to only some or all thedefined groups. When no groups are defined or the allowance for a groupis not specified the, hardware (e.g., the node) is responsible formaking power budgeting decisions for such resources.

Thus, a product implementing the disclosed interface may run “out of thebox” without any specific interface-compliant software stack. Whencompatible software is present, the underlying hardware design uses theprovided information to refine its default power budgeting decisions.The disclosed interface does not depend on the specific hardware powerbudgeting algorithm and may be used in conjunction with traditionalfeedback loop control mechanisms, including low-pass filters,proportional-integral-derivative (PID) controllers, etc.

Accordingly, certain embodiments provide an opt-in design approachwherein if a particular software stack is designed to manage Q resourcesover G groups, the same software stack may be used on different(potentially newer) hardware supporting more resources and/or groups, aslong as the resource addressing is kept consistent across such designs.Unused groups may have no impact and resources not known to the softwaremay be controlled automatically by hardware.

Given that multiple control registers are used to describe an overallset of software requests, certain embodiments provide atomicity toprevent exposing the hardware to partial and possibly inconsistentdefinitions of groups and allowances. When changing allowance values,for example, if a single control register suffices for the allowancedefinition (e.g., eight or less groups supported), a single atomiccontrol register write may be adequate. If two or more independentcontrol register writes are used to define the group allowances, certainembodiments clear out the group definition registers prior to changingthe allowance values, and then restore group information once completed.In certain such embodiments, partial power allowance information has noeffect in the system when no groups are defined.

As another example embodiment providing atomicity, when changing groupdefinitions, if a single control register suffices to describe a node'sgroups, a single atomic control register write may be adequate. If twoor more independent control register writes are used to define thegroups, certain embodiments clear out the group allowance register(s)prior to changing the group definitions. In certain such embodiments,partial group definition information has no effect in the system when noallowances are defined.

FIG. 4 is a flow chart of a method 400 for managing power in a nodeaccording to one embodiment. The node may comprise, for example, aserver or a computing system component such as a microprocessor, anindividual core of a multi-core microprocessor, a memory controller hub(MCH), an input/output controller hub (IOH), a memory device, a networkinterface, the processing node 105 shown in FIG. 1, the CPU sockets 200,300 shown in FIGS. 2 and 3, or any other type of device having aplurality of hardware resources that are capable of being monitored andcontrolled for power consumption.

The method 400 includes providing 410 an interface between a pluralityof hardware resources in a processing node and a power manager, anddefining 412 one or more resource groups exposed to the power managerfor measurement and control through the interface. For each resourcegroup, the method 400 further includes selectively assigning 414 one ormore of the hardware resources within the processing node to theresource group. The method 400 further includes selectively providing416 a power allowance for each resource group.

In certain embodiments, the method 400 also includes monitoring 418 theperformance of the plurality of hardware resources during execution ofan application by the node. A variety of different definitions ofperformance may be used, and performance may be measured at a variety ofdifferent granularities. Thus, performance can be defined through ametric such as rate of instructions retired, floating-point operationsper second (FLOPS), rate of progress, or runtime. Such rates, runtimes,or other performance metrics can be computed at various granularities.For example, the selected performance metric may be measured at shorttime intervals, measured upon an event such as the end of an iteration(for iterative applications), etc.

The method 400 may further include querying 420 whether an optimalgrouping of the plurality of hardware resources has been achieved. Forexample, the monitoring may indicate that processing cores in a firstgroup may improve performance if reassigned to a second group with ahigher power limit. If the optimal grouping has not been achieved, or ifit can be improved, the method 400 may return to selectively assigning414, for each resource group, one or more of the hardware resourceswithin the processing node to the resource group. If, based on currentoperations, the optimal grouping has been achieved, then the method 400also queries 422 whether an optimal power distribution among theresource groups has been achieved. If measurements provided through theinterface determine that the optimal power distribution has not beenreached (or can be further improved), then the method 400 includesredistributing 424 power among the resource groups (e.g., using the“universal coin” to indicate a percentage of a total power limitassigned to each group).

The method 400 also includes querying 426 whether there is a systemchange, application change, power cap change, or selected performancemetric change. If not, the method 400 continues monitoring 418 theperformance of the hardware resources. If, on the other hand, there is adetermined system change, application change, power cap change, orselected performance metric change, the method 400 repeats by againdefining 410 one or more resource groups exposed to the power managerfor measurement and control through the interface, selectively assigning414 hardware resources to the resource groups, and selectively providing416 a power allowance to each resource group.

Example Embodiments

The following are examples of further embodiments. Examples may includesubject matter such as a method, means for perming acts of the method,at least one machine-readable medium including instructions that, whenperformed by a machine cause the machine to perform acts of the method,or of an apparatus or system for rendering augmented video according tothe embodiments and examples described herein.

Example 1 is an apparatus including a node comprising a plurality ofhardware resources, and an interface between the plurality of hardwareresources of the node and a power manager. The interface compriseslogic, at least a portion of which includes hardware. The logic isconfigured to define one or more resource groups to expose to the powermanager for power measurement and control, assign the plurality ofhardware resources to the one or more resource groups; and provide apower allowance to each resource group.

Example 2 includes the subject matter of Example 1, wherein theplurality of hardware resources comprise a plurality of processing coresand at least one non-core resource.

Example 3 includes the subject matter of any of Examples 1-2, whereinthe at least one non-core resource comprises one or more resourcecomprising an interconnect fabric, an input/output device, a memorydevice, and an integrated network interrupt controller (NIC).

Example 4 includes the subject matter of any of Examples 1-3, whereinthe interface comprises a plurality of resource group control registers,and wherein to define the one or more resource groups the interface isconfigured to store resource identification values in one or more of theresource group control registers configured to be exposed to the powermanager.

Example 5 includes the subject matter of Example 4, wherein theinterface is further configured to store a null or zero value in one ormore of the resource group control registers to remove a resource groupfrom exposure to the power manager.

Example 6 includes the subject matter of Example 4, wherein each of theplurality of hardware resources is uniquely associated in the node withone of the resource identification values, and wherein the interface isconfigured to store each resource identification value within one of theresource group control registers corresponding to a selected resourcegroup to which an associated hardware resource is assigned.

Example 7 includes the subject matter of any of Examples 1-6, whereinthe interface comprises one or more power allowance control register,and wherein the interface is configured to store a power allowance valuein the one or more power allowance control register for each of the oneor more resource groups.

Example 8 includes the subject matter of Example 7, wherein the powerallowance value comprises a total power limit for the correspondingresource group.

Example 9 includes the subject matter of Example 7, wherein the powerallowance value comprises a power bias indication of a total power forthe node distributed across the one or more resource groups.

Example 10 includes the subject matter of Example 9, wherein the powerbias indication specifies a percentage of the total power allocated toeach of the one or more resource groups.

Example 11 is a method including providing an interface between aplurality of hardware resources in a processing node and a powermanager, and defining one or more resource groups to expose to the powermanager for measurement and control through the interface. For eachresource group, the method includes selectively assigning one or more ofthe hardware resources within the processing node to the resource group.The method also includes selectively providing a power allowance foreach resource group.

Example 12 includes the subject matter of Example 11, wherein definingthe one or more resource groups comprises writing resourceidentification values in one or more resource group control registersexposed to the power manager.

Example 13 includes the subject matter of Example 12, wherein the methodfurther comprises writing a null or zero value in one or more of theresource group control registers to remove a resource group fromexposure to the power manager.

Example 14 includes the subject matter of Example 12, wherein each ofthe plurality of hardware resources is uniquely associated in theprocessing node with one of the resource identification values, andwherein the method further comprises writing each resourceidentification value within one of the resource group control registerscorresponding to a selected resource group to which an associatedhardware resource is assigned.

Example 15 includes the subject matter of any of Examples 11-14, furthercomprising writing a power allowance value in one or more powerallowance control register for each of the one or more resource groups.

Example 16 includes the subject matter of Example 15, further comprisingdetermining the power allowance value as universal coin whereinincreasing a first portion of a total power limit for the processingnode allocated to a first resource group proportionately decreases asecond portion of the total power limit allocated to a second resourcegroup.

Example 17 includes the subject matter of any of Examples 11-16, whereinthe method further includes monitoring a performance of the one or moreresource groups during an execution of a software application by theprocessing node, and based on the performance, reassigning at least oneof the plurality of hardware resources from a first resource group to asecond resource group.

Example 18 includes the subject matter of any of Examples 11-16, whereinthe method further includes monitoring a performance of the one or moreresource groups during an execution of a software application by theprocessing node, and based on the performance, redistributing a powerlimit for the processing node among the one or more resource groups.

Example 19 is at least one computer-readable storage medium havingstored thereon instructions that, when executed by one or moreprocessors, cause the one or more processors to perform operationscomprising: defining one or more resource groups for measurement andcontrol of a plurality of hardware resources in a processing nodethrough an interface; for each resource group, selectively assigning oneor more of the hardware resources within the processing node to theresource group; and selectively providing a power allowance for eachresource group.

Example 20 includes the subject matter of Example 19, wherein definingthe one or more resource groups comprises writing resourceidentification values in one or more resource group control registers.

Example 21 includes the subject matter of Example 20, wherein each ofthe plurality of hardware resources is uniquely associated in theprocessing node with one of the resource identification values, andwherein the operations further comprise writing each resourceidentification value within one of the resource group control registerscorresponding to a selected resource group to which an associatedhardware resource is assigned.

Example 22 includes the subject matter of any of Examples 19-21, whereinthe operations further comprise writing a power allowance value in oneor more power allowance control register for each of the one or moreresource groups.

Example 23 includes the subject matter of Example 22, wherein theoperations further comprise determining the power allowance value asuniversal coin wherein increasing a first portion of a total power limitfor the processing node allocated to a first resource groupproportionately decreases a second portion of the total power limitallocated to a second resource group.

Example 24 includes the subject matter of any of Examples 19-23, whereinthe operations further comprise: monitoring a performance of the one ormore resource groups during an execution of a software application bythe processing node; and based on the performance, reassigning at leastone of the plurality of hardware resources from a first resource groupto a second resource group.

Example 25 includes the subject matter of any of Examples 19-23, whereinthe operations further comprise: monitoring a performance of the one ormore resource groups during an execution of a software application bythe processing node; and based on the performance, redistributing apower limit for the processing node among the one or more resourcegroups.

Example 26 is a method including defining one or more resource groups toexpose to a power manager for power measurement and control, assigning aplurality of hardware resources to the one or more resource groups, andproviding a power allowance to each resource group.

Example 27 includes the subject matter of Example 26, wherein theplurality of hardware resources comprise a plurality of processing coresand at least one non-core resource in a node.

Example 28 includes the subject matter of any of Examples 26-27, whereinthe at least one non-core resource comprises one or more resourcecomprising an interconnect fabric, an input/output device, a memorydevice, and an integrated network interrupt controller (NIC).

Example 29 includes the subject matter of any of Examples 26-28, whereindefining the one or more resource groups comprises storing resourceidentification values in one or more resource group control registersconfigured to be exposed to the power manager.

Example 30 includes the subject matter of Example 29, wherein the methodfurther comprises storing a null or zero value in one or more of theresource group control registers to remove a resource group fromexposure to the power manager.

Example 31 includes the subject matter of Example 29, wherein each ofthe plurality of hardware resources is uniquely associated with one ofthe resource identification values, the method further comprisingstoring each resource identification value within one of the resourcegroup control registers corresponding to a selected resource group towhich an associated hardware resource is assigned.

Example 32 includes the subject matter of any of Examples 26-31, whereinthe method further comprises storing a power allowance value in one ormore power allowance control register for each of the one or moreresource groups.

Example 33 includes the subject matter of Example 32, wherein the powerallowance value comprises a total power limit for the correspondingresource group.

Example 34 includes the subject matter of Example 32, wherein the powerallowance value comprises a power bias indication of a total power forthe node distributed across the one or more resource groups.

Example 35 includes the subject matter of Example 34, wherein the powerbias indication specifies a percentage of the total power allocated toeach of the one or more resource groups.

Example 36 is an apparatus comprising means to perform a method as inany of Examples 11-18 or 26-35.

Example 37 is a machine readable storage including machine-readableinstructions to implement a method as in any of Examples 11-18 or 26-35.

The above description provides numerous specific details for a thoroughunderstanding of the embodiments described herein. However, those ofskill in the art will recognize that one or more of the specific detailsmay be omitted, or other methods, components, or materials may be used.In some cases, well-known features, structures, or operations are notshown or described in detail.

Furthermore, the described features, operations, or characteristics maybe arranged and designed in a wide variety of different configurationsand/or combined in any suitable manner in one or more embodiments. Thus,the detailed description of the embodiments of the systems and methodsis not intended to limit the scope of the disclosure, as claimed, but ismerely representative of possible embodiments of the disclosure. Inaddition, it will also be readily understood that the order of the stepsor actions of the methods described in connection with the embodimentsdisclosed may be changed as would be apparent to those skilled in theart. Thus, any order in the drawings or Detailed Description is forillustrative purposes only and is not meant to imply a required order,unless specified to require an order.

The term “coupled” may be used herein to refer to any type ofrelationship, direct or indirect, between the components in question,and may apply to electrical, mechanical, fluid, optical,electromagnetic, electromechanical or other connections. In addition,the terms “first”, “second”, etc. might be used herein only tofacilitate discussion, and carry no particular temporal or chronologicalsignificance unless otherwise indicated.

Any reference in this specification to “one embodiment,” “anembodiment,” “example embodiment,” etc., means that a particularfeature, structure, or characteristic described in connection with theembodiment is included in at least one embodiment of the invention. Theappearances of such phrases in various places in the specification arenot necessarily all referring to the same embodiment. Further, when aparticular feature, structure, or characteristic is described inconnection with any embodiment, it is submitted that it is within thepurview of one skilled in the art to affect such feature, structure, orcharacteristic in connection with other ones of the embodiments.

Various embodiments may be implemented using hardware elements, softwareelements, and/or a combination of both. Examples of hardware elementsmay include processors, microprocessors, circuits, circuit elements(e.g., transistors, resistors, capacitors, inductors, and so forth),integrated circuits, application specific integrated circuits (ASIC),programmable logic devices (PLD), digital signal processors (DSP), fieldprogrammable gate array (FPGA), logic gates, registers, semiconductordevice, chips, microchips, chip sets, and so forth. Examples of softwaremay include software components, programs, applications, computerprograms, application programs, system programs, machine programs,operating system software, middleware, firmware, software modules,routines, subroutines, functions, methods, procedures, softwareinterfaces, application program interfaces (API), instruction sets,computing code, computer code, code segments, computer code segments,words, values, symbols, or any combination thereof.

One or more aspects of at least one embodiment may be implemented byrepresentative instructions stored on a machine-readable medium whichrepresents various logic within the processor, which when read by amachine causes the machine to fabricate logic to perform the techniquesdescribed herein. Such representations, known as “IP cores” may bestored on a tangible, machine readable medium and supplied to variouscustomers or manufacturing facilities to load into the fabricationmachines that actually make the logic or processor.

Although embodiments have been described with reference to a number ofillustrative embodiments thereof, it should be understood that numerousother modifications and embodiments can be devised by those skilled inthe art that will fall within the spirit and scope of the principles ofthis disclosure. More particularly, various variations and modificationsare possible in the component parts and/or arrangements of the subjectcombination arrangement within the scope of the disclosure, the drawingsand the appended claims. In addition to variations and modifications inthe component parts and/or arrangements, alternative uses will also beapparent to those skilled in the art. The scope of the present inventionshould, therefore, be determined only by the following claims.

The invention claimed is:
 1. An apparatus comprising: an interface tocommunicate with a plurality of hardware resources on different powerdomains and a power manager; logic in communication with the interface,the logic to: define one or more resource groups to expose to the powermanager for power measurement and control; assign the plurality ofhardware resources to the one or more resource groups, wherein at leastone resource group comprises a first hardware resource on a first powerdomain and a second hardware resource on a second power domain forming arelative power domain; provide a power allowance to each resource group,wherein the power allowance for a resource group associated with therelative power domain controls both the first hardware resource on thefirst power domain and the second hardware resource on the second powerdomain; and selectively and dynamically reassign, during operation, theplurality of hardware resources to the one or more resource groups, bymanipulating a plurality of resource group control registers to storeresource identification values uniquely associated with each of theplurality of hardware resources within one of the plurality of resourcegroup control registers corresponding to an assigned resource group. 2.The apparatus of claim 1, wherein the plurality of hardware resourcescomprises a plurality of processing cores and at least one non-coreresource.
 3. The apparatus of claim 2, wherein the at least one non-coreresource comprises one or more resource comprising an interconnectfabric, an input/output device, a memory device, and an integratednetwork interrupt controller (NIC).
 4. The apparatus of claim 1, furthercomprising the plurality of resource group control registers, andwherein to define the one or more resource groups the logic isconfigured to store resource identification values in the plurality ofresource group control registers configured to be exposed to the powermanager.
 5. The apparatus of claim 4, wherein the logic is furtherconfigured to store a null or zero value in the plurality of resourcegroup control registers to remove a resource group from exposure to thepower manager.
 6. The apparatus of claim 4, wherein each of theplurality of hardware resources is uniquely associated with one of theresource identification values, and wherein the logic is configured tostore each resource identification value within one of a resource groupcontrol registers of the plurality corresponding to a selected resourcegroup to which an associated hardware resource is assigned.
 7. Theapparatus of claim 1, further comprising one or more power allowancecontrol register, and wherein the logic is configured to store a powerallowance value in the one or more power allowance control register foreach of the one or more resource groups.
 8. The apparatus of claim 7,wherein the power allowance value comprises a total power limit for acorresponding resource group.
 9. The apparatus of claim 7, wherein thepower allowance value comprises a power bias indication of a total powerfor a node distributed across the one or more resource groups.
 10. Theapparatus of claim 9, wherein the power bias indication specifies apercentage of the total power allocated to each of the one or moreresource groups.
 11. A method comprising: defining one or more resourcegroups to expose to a power manager for power measurement and control;assigning one or more hardware resources to the one or more resourcegroups, wherein at least one resource group comprises a first hardwareresource on a first power domain and a second hardware resource on asecond power domain forming a relative power domain; providing a powerallowance to each resource group, wherein the power allowance for aresource group associated with the relative power domain controls boththe first hardware resource on the first power domain and the secondhardware resource on the second power domain; and selectively anddynamically reassigning, during operation, the one or more hardwareresources to the one or more resource groups, by manipulating aplurality of resource group control registers to store resourceidentification values uniquely associated with each of the one or morehardware resources within one of the plurality of resource group controlregisters corresponding to an assigned resource group.
 12. The method ofclaim 11, wherein defining the one or more resource groups compriseswriting resource identification values in one or more resource groupcontrol registers exposed to the power manager.
 13. The method of claim12, further comprising writing a null or zero value in the one or moreof resource group control registers to remove a resource group fromexposure to the power manager.
 14. The method of claim 12, wherein eachof the one or more hardware resources is uniquely associated in aprocessing node with one of the resource identification values, andwherein the method further comprises writing each resourceidentification value within one of the resource group control registerscorresponding to a selected resource group to which an associatedhardware resource is assigned.
 15. The method of claim 11, furthercomprising writing a power allowance value in one or more powerallowance control register for each of the one or more resource groups.16. An apparatus comprising: an interface to communicate with aplurality of hardware resources; a plurality registers including groupcontrol registers and power allowance control registers; and logic incommunication with the interface, the group control registers, and thepower allowance control registers, the logic to: write resource groupvalues to one or more of the group control registers to define one ormore resource groups, wherein each resource group clusters hardwareresources based on a current operating behavior; write power budgetvalues to the power allowance control registers to define a powerdistribution among the one or more resource groups; and selectively anddynamically reassign, during operation, one of the plurality of hardwareresources from a first resource group to a second resource group toimprove performance, by manipulating the group control registers tostore resource identification values uniquely associated with each ofthe plurality of hardware resources within one of the resource groupcontrol registers corresponding to an assigned resource group.
 17. Theapparatus of claim 16, wherein the plurality of hardware resourcescomprises a plurality of elements on a first power domain and at leastone element on a second power domain.
 18. The apparatus of claim 16,wherein each of the plurality of hardware resources is uniquelyassociated with one of the resource identification values, and whereinthe interface is configured to store each resource identification valuewithin one of the resource group control registers corresponding to aselected resource group to which an associated hardware resource isassigned.
 19. The apparatus of claim 16, wherein the power allowancevalue comprises a total power limit for the corresponding resourcegroup.
 20. The apparatus of claim 16, wherein the power allowance valuecomprises a power bias indication of a total power for a nodedistributed across the one or more resource groups.